A Methodology for Information Theoretic Feature Extraction

We discuss an unsupervised feature extraction method which is driven by an information theoretic based criterion: mutual information. While information theoretic signal processing has been examined by many authors the method presented here is more closely related to the approaches of Linsker (1988,1990). Bell and Sejnowski (1995), and Viola et a1 (1996). The method we discuss differs from previous work in several aspects. It is extensible to a fiedforward multi-layer perceptmn with an arbitrary number of layers. No assumptions are ma& about the underlying PDF of the input space. It exploits a pmperty of entropy coupled with a saturating nonlinearity resulting in a method for entropy manipulation with computational complexity proportional to the number of data samples squared. This repments a significant computationaI savings overprevious methods (viola et al, 1996). As mutual infomation is a function of two entropy terms, the method for entropy manipulation can be directly applied to the mutual information as well.